A Testing Framework for AI Linguistic Systems (testFAILS)

This paper presents an innovative testing framework, testFAILS, designed for the rigorous evaluation of AI Linguistic Systems (AILS), with particular emphasis on various iterations ChatGPT. Leveraging orthogonal array coverage, this framework provides a robust mechanism assessing systems, addressing critical question, “How should be evaluated?” While Turing test has traditionally been benchmark evaluation, it is argued that current, publicly available chatbots, despite their rapid advancements, have yet to meet standard. However, pace progress suggests achieving Turing-test-level performance may imminent. In interim, need effective and methodologies remains paramount. Ongoing research already validated several versions ChatGPT, comprehensive latest models, including ChatGPT-4, Bard, Bing Bot, LLaMA PaLM 2 currently being conducted. The testFAILS adaptable, ready evaluate new chatbot as they are released. Additionally, APIs tested applications developed, one them AIDoctor, presented in paper, which utilizes ChatGPT-4 model Microsoft Azure technologies.